Efficient Image Demosaicing and Local Contrast Enhancement

ABSTRACT

An exemplary method includes a demosaic module receiving a frame of raw pixels having values in a first color space, generating vertical and horizontal color difference signals using information from the frame of raw pixels, generating reconstructed color pixel values for raw pixels that are missing captured color pixel values of a first color component of the first color space, transforming the captured and reconstructed color pixel values of the first color component to brightness pixel values in a second color space, and transforming the vertical and horizontal color difference signals into chrominance pixel values in the second color space.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/917,483 filed Jun. 30, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/239,489 filed Jan. 3, 2019 and issued as U.S.Pat. No. 10,733,703, which application is a continuation of U.S. patentapplication Ser. No. 14/451,621 filed Aug. 5, 2014 and issued as U.S.Pat. No. 10,210,599, which claims priority to and the benefit of U.S.Provisional Application No. 61/864,090 filed Aug. 9, 2013, each of whichis incorporated herein by reference in its entirety.

BACKGROUND Field of Invention

Aspects of this invention are related to color imaging in surgicalsystems, and are more particularly related to demosaicing a frame ofpixels.

Related Art

Many color cameras capture images using a single sensor array. Thecamera includes a color filter array and an image capture sensor. Thecolor filter array filters the incoming light so that each pixel of theimage capture sensor receives only one of the primary colors. Spatialresolution is sacrificed in view of the compact size of the singlesensor array and the ability to generate full resolution images from thecaptured pixels.

A commonly used color filter array is referred to as a Bayer patterncolor filter array. The Bayer pattern color filter array is described,for example, in U.S. Pat. No. 3,971,065, which is incorporated herein byreference.

FIG. 1A is a representation of part of a Bayer pattern color filterarray. The Bayer pattern color filter array has a mosaic pattern. Eachpixel in the Bayer pattern color filter array passes primarily a singleprimary color in a color space. In a color space with red, green, andblue color components, the pixels in the array selectively pass greenlight, red light, and blue light to the image capture sensor. Typically,the Bayer pattern color filter array is a mosaic with one-quarter of thetotal number of pixels being red (R) filters, one-half of the totalnumber of pixels being green (G) filters; and one-quarter of the totalnumber of pixels being blue (B) filters.

In the image acquired by the image capture sensor, no color componenthas the full resolution of the image capture sensor, i.e., the totalnumber of pixels captured by the image capture sensor. As shown in FIG.1B, the captured green pixels G are only one-half of the total number ofpixels captured by the image capture sensor. In FIGS. 1C and 1D, thecaptured red pixels and blue pixels are each only one-quarter of thetotal number of pixels captured by the image capture sensor. To obtain acomplete full resolution set of pixels for each of the color components,a process called demosaicing is commonly used. Demosaicing is a processused to reconstruct a full color image from the color samples that areoutput from an image capture sensor overlaid with a color filter array.

Since humans are sensitive to edge structures in an image, many adaptivedemosaicing methods try to avoid interpolating across edges. Someexamples of these methods are presented in U.S. Pat. No. 5,629,734(disclosing “Adaptive Color Plane Interpolation in Single Sensor ColorElectronic Camera”) and L. Zhang and X. Wu, “Color Demosaicing TheDirectional Linear Minimum Mean Square Error Estimation,” IEEE Trans. onImaging Processing, 2005.

U.S. Pat. No. 5,629,734 describes an adaptive interpolation demosaicingprocess that utilizes a second order Laplacian filter. The filter usesthe absolute value of a second order blue-red color gradient plus theabsolute value of a first order green color difference to select an edgedirection in the adaptive interpolation demosaicing process.

Each demosaicing process tries to improve on one or more of the factorsthat are important to demosaicing. A few factors considered in selectinga demosaicing process include: 1) reconstruction quality in terms ofedges, 2) reconstruction quality in terms of color artifacts, 3) processcomplexity, and 4) reconstruction quality under noise. Unfortunately,many of the demosaicing processes that perform well with respect to thefirst, second, and fourth factors are complex, and it is challenging toimplement them, without expensive hardware, in a surgical system thatrequires a real-time video display.

SUMMARY

In one aspect, an efficient demosaicing process receives a frame ofpixels. The frame of pixels was captured in an image capture unitincluding a red-green-blue Bayer color filter array. The efficientdemosaicing process generates a plurality of color difference signalsfrom pixels in the frame and then filters each of the color differencesignals in the plurality of color difference signals. The efficientdemosaicing process constructs a full resolution frame of green pixelsby adaptively generating missing green pixels from the filtered colordifference signals. The adaptive generation process includes selectingthe filtered color difference signal used with a local edge estimatorthat is specific to each pixel and so different pixels can havedifferent local edge estimators.

In one aspect, the generation of the plurality of color differencesignals includes generating a series expansion of each missing pixel ateach location in the frame. The series expansion is an estimated valueof the missing pixel and so an estimated pixel is generated. In oneaspect, the series expansion is used to generate an estimated horizontalpixel and an estimated vertical pixel.

The generation of a plurality of color difference signals includesgenerating, for one of a red pixel and a blue pixel, a difference of anestimated green pixel and the one of the red pixel and the blue pixel.The generation of a plurality of color difference signals also includesgenerating, for a green pixel, a difference of the value of the greenpixel and one of an estimated red pixel and an estimated blue pixel.

In another aspect, the efficient demosaicing process is enhanced byincluding local contrast enhancement. In this method, a frame of pixelsis received. The efficient demosaicing process with local contrastenhancement generates a plurality of color difference signals frompixels in the frame, and then filters each of the color differencesignals in the plurality of color difference signals. The efficientdemosaicing process constructs a full resolution frame of green pixelsby adaptively generating missing green pixels from the filtered colordifference signals. The adaptive generation process includes selectingthe filtered color difference signal used in the generating with a localedge estimator that is specific to each pixel and so different pixelscan have different local edge estimators.

The efficient demosaicing process with local contrast enhancementtransforms a green pixel in the full resolution frame of green pixelsinto a brightness pixel by scaling the green pixel with a scale factor.The brightness pixel is in a color space different from thered-green-blue color space. The efficient demosaicing process with localcontrast enhancement also transforms the plurality of filtered colordifference signals into chrominance pixels in the color space differentfrom a red-green-blue color space.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to 1D are different illustrations of the mosaic pixel layout inframe of pixels captured in an image capture unit with a single imagecapture sensor and a red-green-blue color filter array.

FIG. 2 is schematic diagram of a teleoperated surgical system includingan imaging processing system that includes demosaicing modules thatperform an efficient demosaicing process that in one aspect includeslocal contrast enhancement.

FIG. 3 is a process flow diagram of the efficient demosaicing processperformed by the demosaicing modules of FIG. 2.

FIG. 4 illustrates a portion of a full resolution frame of pixelscreated by the process of FIG. 3, in one aspect.

FIG. 5A is a schematic illustration of a block of raw pixel data withina frame received from an image capture unit with a Bayer color filterarray.

FIG. 5B illustrates the estimated pixels at each location in the blockof FIG. 5A and the captured pixel at each location.

FIG. 5C illustrates the color difference signals generated at eachlocation in the block of FIG. 5A.

FIG. 6 is a schematic illustration of a low pass filter that is used, inone aspect, to filter the color difference signals.

FIG. 7 illustrates the filtered color difference signals generated ateach location in the block of FIG. 5A.

FIG. 8 is a more detailed process flow diagram of the adaptivereconstruction of green pixels process in FIG. 3.

FIG. 9 illustrates the pixels and the filtered color difference signalsat each location in the block of FIG. 5A following completion of theadaptive reconstruction of green pixels process in FIG. 8.

FIG. 10 is a more detailed process flow diagram of the reconstruct redand blue pixels process of FIG. 3.

FIGS. 11A to 11D are representations of information used in reconstructred and blue pixels process of FIG. 3.

FIGS. 12A to 12D show the pixel color layout and coordinate positionused in the series expansion to generate the estimated pixel values.

FIG. 13 is a process flow diagram of the efficient demosaicing processwith local contrast enhancement performed by the demosaicing modules ofFIG. 2.

FIG. 14 is a more detailed process flow diagram of the generatechrominance pixels in a second color space process of FIG. 13.

FIGS. 15A to 15D are representations of information used in generatingthe color difference signals in the process of FIG. 14.

In the drawings, for single digit figure numbers, the first digit of areference number indicates the figure number in which an element withthat reference number first appeared. For double-digit figure numbers,the first two digits of a reference number indicate the figure number inwhich an element with that reference number first appeared.

DETAILED DESCRIPTION

Aspects of this invention are used in a surgical system, e.g., the daVinci® Surgical Robotic System commercialized by Intuitive Surgical,Inc. of Sunnyvale, Calif., when color imaging is used. (da Vinci® is aregistered trademark of Intuitive Surgical, Inc. of Sunnyvale, Calif.).An efficient demosaicing process turns raw scenes from a single-chip CFA(color filter array) sensor to color scenes that make up a videosequence in the surgical system. In one aspect, the efficientdemosaicing process is combined with local contrast enhancement in anefficient way that is particularly effective for surgical scenes.

The efficient demosaicing process has good reconstruction quality interms of edges, good reconstruction quality in terms of color artifacts,and good reconstruction quality under noise. In one aspect, theefficient demosaicing process is implemented in hardware so that thedelay in the image processing introduced by the demosaicing process isreduced significantly in comparison to prior art software based methods.

As in any demosaicing process, the efficient demosaicing processgenerates full resolution sets of red, green, and blue pixels from thepartial sets of red, green, and blue pixels captured by the imagecapture sensor. The image capture sensor captures a monochromatic image.However, since the locations in the image capture sensor that correspondto the red, green, and blue pixels are known, each captured pixel isassigned the appropriate color and processed as a pixel of that color.

An example of the efficient demosaicing process is presented for animage capture unit that includes a Bayer pattern color filter array andan image capture sensor. It is known that the Bayer pattern color filterarray has the best spatial arrangement for a three-color color filterarray. See, for example, U.S. Pat. No. 3,971,056 (disclosing “ColorImaging Array”). The demosaicing process relies upon well-knownassumptions that the different red, green, and blue color components arecorrelated. See U.S. Pat. No. 4,642,678 (disclosing “Signal ProcessingMethod and Apparatus for Producing Interpolated Chromatic Values in aSampled Color Image Signal”) and L. Zhang and X. Wu, “Color DemosaicingThe Directional Linear Minimum Mean Square Error Estimation,” IEEETrans. on Imaging Processing, 2005.

In particular, the efficient demosaicing method follows the assumptionsthat the color-difference signals between a green pixel and a red pixelat the same location and between a green pixel and the blue pixel atthat location are low pass signals and that the second order gradientsof different color components are similar. Edge-adaptiveness is alsobuilt in the efficient demosaicing process to better handle edgereconstruction. As a result, the efficient demosaicing process ischeaper to implement than the statistical demosaicing methods, butproduces similar images that are better than popular adaptiveinterpolation processes in handling color artifacts and noise.

In one aspect, logically, the efficient demosaicing method includes tworeconstruction processes. First, missing green pixels are reconstructedafter estimating green-red color difference signals and green-blue colordifference signals that are used in the reconstruction. Second, missingred and blue pixels are constructed using the color difference signals.Thus, the two reconstruction processes create a full resolution frame ofpixels. A full resolution frame of pixels has a red pixel, a greenpixel, and a blue pixel at each location in the frame. In one aspect,the full resolution frame of pixels is sent to a display unit fordisplay.

Contrast enhanced images make it easier for surgeons to differentiatecritical structures, such as blood vessels from surrounding tissues,when viewing a surgical site on a display of a surgical system. Insurgical imaging, the predominant color is red for both tissues andblood vessels, making it hard for surgeons to differentiate criticalstructures from surrounding tissues. In addition, the red colorcomponent has lower contrast compared to green and blue color componentsdue to tissue scattering of light that is especially true in close-rangeimaging. This makes it even harder to differentiate fine structures fromsurrounding tissues. Typically, compared to the blue component, thegreen component has a better signal-to-noise ratio due to sensorsensitivity.

In one aspect, the efficient demosaicing process includes local contrastenhancement. When the efficient demosaicing process includes contrastenhancement, the process is sometimes referred to as an enhanceddemosaicing process. The key idea of enhancing image contrast is tobuild and boost a brightness component, e.g., a luminance component Y,from the green component, and build chromatic components, e.g., UV (orCb, Cr) components, from all of the red, green, and blue colorcomponents.

In this aspect, logically, the enhanced demosaicing method includesthree processes. First, missing green pixels are reconstructed afterestimating green-red color difference signals and green-blue colordifference signals that are used in the reconstruction. Second, each ofgreen pixels is transformed to a brightness pixel by scaling the greenpixel. Third, the missing red and blue pixels are not reconstructed.Instead, the green-red color difference signals and the green-blue colordifference signals are transformed into chromatic component pixels.Following completion of the reconstruction, a full resolution frame ofpixels is obtained, e.g., a frame with a Y pixel, a U pixel, and a Vpixel at each location in the frame. In one aspect, this full resolutionframe of pixels is sent to a display unit for display.

FIG. 2 is a schematic illustration of part of a teleoperated surgicalsystem 200, such as a minimally invasive surgical system, e.g., the daVinci® Surgical System. There are other components, parts, cables etc.associated with system 200 and with the da Vinci® Surgical System, butthese are not illustrated in FIG. 2 to avoid detracting from thedisclosure.

In this aspect, surgical system 200 includes an illuminator 215 (FIG. 2)that is coupled to a stereoscopic endoscope 212. In one aspect,illuminator 215 generates white light.

In one example, three color components make up white light illumination,i.e., white light includes first color component C1, second colorcomponent C2, and third color component C3. Each of the three-colorcomponents C1, C2, C3 is a different visible color component, e.g., ared color component, a green color component, and a blue colorcomponent.

The use of three visible color components C1, C2, C3 to make up whitelight illumination is illustrative of a plurality of such illuminationcomponents and is not intended to be limiting. Also, first, second, andthird are used only as adjectives to distinguish between the differentcolor components and are not intended to imply that any wavelength orfrequency spectrum is associated with a particular color component.

In one aspect, illuminator 215 includes a source for each of thedifferent visible color illumination components in the plurality ofvisible color illumination components of white light. For ared-green-blue implementation, in one example, the sources are lightemitting diodes (LEDs), a red LED, two green LEDs and a blue LED.

The use of LEDs in illuminator 215 is illustrative only and is notintended to be limiting. Illuminator 215 could also be implemented withmultiple laser sources instead of LEDs, for example. Alternatively,illuminator 215 could use a Xenon lamp with an elliptic back reflectorand a band pass filter coating to create broadband white illuminationlight. The use of a Xenon lamp also is illustrative only and is notintended to be limiting. For example, a high-pressure mercury arc lamp,other arc lamps, or other broadband light sources may be used.

The use of an illuminator that is removed from endoscope 212 is alsoillustrative and not intended to be limiting. In some aspects, theilluminator could be mounted on or in the distal end of endoscope 212,for example.

In the aspect of FIG. 2, the light from illuminator 215 is directed intoan illumination channel 216. The light passes through illuminationchannel 216 to another illumination channel in stereoscopic endoscope212 that in turn directs the light to surgical site 203. Thus, in thisaspect, surgical site 203 is illuminated by the white light.

The reflected white light from surgical site 203 (FIG. 2) is passed bythe stereoscopic optical channels—a left channel and a right channel—inendoscope 212 to image capture system 220. Image capture system 220, inthis aspect, includes a left image capture unit 221 that includes afirst single-chip image capture sensor 225 with a first Bayer colorfilter array 223 and a right image capture unit 222 that includes asecond single-chip image capture sensor 226 with a second Bayer colorfilter array 224. Each of Bayer color filter arrays 223 and 224 haspixel layout 100 (FIG. 1A).

Left image capture unit 221 captures a left frame and right imagecapture unit 222 captures a right frame. Both frames have the pixellayout of FIG. 1A. Camera control units 231 and 232 may be a single dualcontroller unit, or may be two separate controller units. Camera controlunits 231 and 232 receive signals from a system process module 263 thatcontrols gains, controls capturing frames, and controls transferringcaptured frames to image processing system 240. Typically, systemprocess module 263 with camera control units 231 and 232 controls thefrequency at which the frames are captured. System process module 263represents the various controllers including the vision systemcontrollers in system 200.

Image processing system 240 includes, in one aspect, left demosaicmodule 243, left image processing system 241, right demosaic module 244,and right image processing system 242. In the following example, theprocessing performed by left image processing system 241 and by rightimage processing system 242 is the same. Similarly, the processingperformed by left demosaic module 243 and by right demosaic module 244is the same.

Thus, in the following description when an aspect of the efficientdemosaicing process is described the process applies to both the leftand the right channels. The description is not repeated for each of theleft and right channels because the configuration and operation of thetwo channels is the same. Also, the processing of a single acquiredcolor component frame is described. The processing performed on thesingle frame is performed repeatedly on each of the captured frames,i.e., a video sequence is generated.

The use of a stereoscopic endoscope with left and right processingchannels also is illustrative only and is not intended to be limiting.The efficient demosaicing process and the efficient demosaicing processwith local contrast enhancement can be used in any system that includesan image capture unit with an image capture sensor and a color filterarray. Also, the inclusion of demosaic modules 243, 244 in imageprocessing system 240 is illustrative only and is not intended to belimited. When demosaic modules 243, 244 are implemented in hardware,demosaic modules 243 and 244 can be included in image capture units 221and 222.

Image processing system 240 can implement a variety of techniques toimprove the quality of the images that are sent to stereoscopic displayunit 250 in the surgeons control console. Optional image processingmodule 260 receives processes images from image processing system 240prior to display on stereoscopic display unit 250. Optional imageprocessing module 260 is equivalent to image processing modules in priorart teleoperated surgical systems and so is not considered in furtherdetail.

Prior to considering efficient demosaicing module 243 further, it ishelpful to understand the coordinate system of frame 100 and of theblocks of pixels described below. Each frame and each block has rows ofpixels and columns of pixels. For example, frame 100 could have 1080rows of 1920 pixels. A horizontal location x of a pixel within a frameis a value on an x-axis and a vertical location y of that pixel isrepresented by a value on the y-axis. The pixel at location (x, y) isthe pixel that is currently being processed and is referred as thecurrent pixel. In the example of a frame with 1920 by 1080 pixels, thevalue of x ranges from 0 to 1919, and the value of y ranges from 0 to1079.

Frame 100 is a frame of raw pixels and has one pixel at each location inthe frame. The efficient demosaicing process generates a new frame ofpixels. The number of locations in frame 100 and the new frame typicallyare the same. However, the new frame has three pixels at each locationin that frame.

A pixel is measured, in one aspect, with a value ranging from zero to255, where zero is no light and 255 is maximum intensity. The range ofvalues used herein is for illustration only and are not intended to belimiting. As is known to those knowledgeable in the field, a pixel witha value ranging from zero to 255 is stored as one byte of data. If morebytes of data are used for a pixel, the number of possible valueschanges.

FIG. 3 is a process flow diagram of an EFFICIENT DEMOSAICING process 300that is performed by demosaic module 243. Sometimes, EFFICIENTDEMOSAICING process 300 is referred to as process 300. In RECEIVECAPTURED FRAME process 301, demosaic module 243, sometimes referred toas module 243, receives a frame of pixels from image capture unit 221.The pixels in the received frame have the pixel layout of frame 100(FIG. 1A) due to Bayer color filter array 223.

In RECONSTRUCT GREEN PIXELS process 310, sometimes referred to asprocess 310, demosaic module 243 reconstructs missing green pixels sothat all locations in frame 100G (FIG. 1B) have a green pixel value. Inframe 100G, green pixels that were captured by image capture sensor 225are represented by “G”. As explained more completely below, GENERATEESTIMATE OF MISSING PIXELS process 302 generates a horizontal estimatedpixel and a vertical estimated pixel for each pixel location in receivedframe 100. Sometimes GENERATE ESTIMATE OF MISSING PIXELS process 302 isreferred to as process 302.

Next GENERATE COLOR DIFFERENCES process 303 generates horizontal andvertical color difference signals, i.e., horizontal and verticalgreen-red color difference signals and horizontal and verticalgreen-blue color difference signals. Sometime these color differencesignals are referred to as raw color difference signals or as colordifferences. After the color difference signals are available, FILTERCOLOR DIFFERENCES process 304 filters each color difference signal togenerate filtered color difference signals, sometimes referred to asfiltered color differences.

At each location missing a green pixel in frame 100G (FIG. 1B), ADAPTIVERECONSTRUCTION OF GREEN PIXELS process 305, sometimes referred to asprocess 305, generates a reconstructed green pixel g for the location.Herein, when it is stated that a reconstructed green pixel g isgenerated, it means that a value of the pixel is generated whichrepresents the intensity of that pixel.

Initially, in one aspect, process 305 generates a horizontal local edgeestimator and a vertical local edge estimator for the missing greenpixel. The relationship between the two local edge estimators is used toselect either one of the horizontal filtered color difference signal andthe vertical filtered color difference signal, or a combination of thehorizontal and filtered color difference signals in reconstructingmissing green pixel. The filtered color difference signal selected isthe one that has the least edge effects. This color difference signal isused in reconstructing the missing green pixel g. Process 305 isadaptive, because the local edge characteristics at each pixel locationare used in determining how to reconstruct the missing green pixel.Specifically, the process adapts to the edge conditions at each pixellocation in frame 100.

When process 305 is completed, each location in the frame has either agreen pixel G or a reconstructed green pixel g. Following completion ofprocess 305, processing transfers to RECONSTRUCT RED AND BLUE PIXELSprocess 311, sometimes referred to as process 311. Process 311reconstructs each missing blue pixel and each missing red pixel.

At each location missing a red pixel in frame 100R (FIG. 1C),RECONSTRUCT RED AND BLUE PIXELS process 311 reconstructs red pixels sothat all these locations have a reconstructed red pixel r. In frame100R, red pixels that were captured by image capture sensor 225 arerepresented by “R,” and are sometimes referred to as original red pixelR. At each location missing a blue pixel in frame 100B (FIG. 1D),process 311 reconstructs blue pixels b so that all these locations havea reconstructed blue pixel b. In frame 100B, blue pixels that werecaptured by image capture sensor 225 are represented by “B,” and aresometimes referred to as original blue pixel B. Note that while the samereference letter is used for pixels of the same color and same type(original or reconstructed), this denotes only the color of the pixeland not the value of the pixel.

RECONSTRUCT RED AND BLUE PIXELS process 311, in one aspect, uses thesame adaptive process to reconstruct the red and blue pixels that wasused to reconstruct the green pixels. In another aspect, for a bluepixel B, process 311 first averages the filtered color differencesignals of the four red pixels R nearest blue pixel B. Process 311 thenadds the average filtered color difference signal to the reconstructedgreen pixel g at the location to obtain the reconstructed red pixel r.Thus, each location with an original blue pixel B has original bluepixel B, a reconstructed green pixel g, and a reconstructed red pixel r.

In this aspect, for a red pixel R, process 311 first averages thefiltered color difference signals of the four blue pixels B nearest redpixel R. Process 311 then adds the average filtered color differencesignals to the reconstructed green pixel g at the location to obtain thereconstructed blue pixel b. Thus, each location with an original redpixel R has original red pixel R, a reconstructed green pixel g, and areconstructed blue pixel b.

In this aspect, for a green pixel G, process 311 first averages thefiltered horizontal color difference signals of the horizontal pixelsnearest green pixel G. Process 311 then adds the average filteredhorizontal color difference signals to green pixel G at the location toobtain one of a reconstructed red pixel r and a reconstructed blue pixelb. The pixel reconstructed depends on the color of the horizontal pixelsadjacent to green pixel G. Process 311 next averages the filteredvertical color difference signals of the vertical pixels nearest greenpixel G. Process 311 then adds the average filtered vertical colordifference signal to green pixel G at the location to obtain the otherof reconstructed red pixel r and reconstructed blue pixel b. Again, thepixel reconstructed depends on the color of the vertical pixels adjacentto green pixel G. Each location with an original green pixel G has areconstructed red pixel r, original green pixel G, and a reconstructedblue pixel b.

Thus, demosaic module 243 reconstructs red, green, and blue pixels sothat each location in frame 100 has not only the originally capturedpixel, but also two reconstructed pixels as illustrated in new frame 400(FIG. 4). Herein, reconstructed green pixels are represented by “g”;reconstructed red pixels are represented by “r”; and reconstructed bluepixels are represented by “b”. While the same letter is used for a pixelat different locations in frame 400, this should be interpreted asindicating all the pixels with the same letter have the same value.Frame 400 is intended to show which pixels are reconstructed at eachlocation.

Locations in frame 400 that have a captured green pixel G now alsoinclude a reconstructed red pixel r and a reconstructed blue pixel b.Locations in frame 400 that have a captured red pixel R now also includea reconstructed green pixel g and a reconstructed blue pixel b.Locations in frame 400 that have a captured blue pixel B now alsoinclude a reconstructed red pixel r and a reconstructed green pixel g.The following description provides greater detail on the reconstructionprocess for each of the reconstructed pixels.

SEND FULL RESOLUTION FRAME process 312 transfers frame 400 to left imageprocessing system 241. Process 300 then starts demosaicing the nextframe received from image capture unit 221.

Now, each of the processes in EFFICIENT DEMOSAICING process 300 isconsidered in further detail. In one aspect, process 300 of demosaicmodule 243 is implemented in hardware. In another aspect, module 243includes computer program instructions that are stored in a memory andare executed on a processor. However, this is illustrative only and isnot intended to be limiting. Demosaic module 243 may be implemented inpractice by any combination of hardware, software that is executed on aprocessor, and firmware. Also, functions performed by demosaic module,as described herein, may be performed by one unit, or divided up amongdifferent components, each of which may be implemented in turn by anycombination of hardware, software that is executed on a processor, andfirmware. When divided up among different components, the components maybe centralized in one location or distributed across system 200 fordistributed processing purposes.

FIG. 5A is a schematic illustration of a block of raw pixel data 550within a frame received from an image capture unit with a Bayer colorfilter array. Within block 550, block 500 is considered in the followingmore detailed description of process 300. The processing of pixels inblock 500, with the exception of the center pixel, requires raw pixeldata in block 550 that is outside block 500. Block 550 includessufficient raw pixel data that each pixel in block 500 can be processedas described below.

The outmost pixels of a received frame, i.e., the pixels in the two topand bottom rows of the received frame and the pixels in the two leftmost and right most columns in the received frame can be processed in anumber of ways. The pixels values in the reconstructed pixels inadjacent rows and columns could be used in the outermost pixels, or rawpixel data can be reflected to create a border of pixels for thereceived frame and the outmost pixels processed as described below.

In RECONSTRUCT GREEN PIXELS process 310, GENERATE ESTIMATE OF MISSINGPIXELS process 302, sometimes referred to as process 302, generates twopixels, a horizontal estimated pixel and a vertical estimated pixel foreach pixel location. In one aspect, process 302 uses a second orderseries expansion to generate the estimated pixel values at a location.For example, when the current pixel is a green pixel G at location (x,y), process 302 uses the second order series expansion to generateestimated values for the red and blue pixels at location (x, y). Whenthe current pixel is a red pixel R at location (x, y), process 302 usesthe second order series expansion to generate estimated values for agreen pixel at location (x, y). When the current pixel is a blue pixel Bis at location (x, y), process 302 uses the second order seriesexpansion to generate estimated values for a green pixel at location (x,y).

For each green pixel in a row of blue and green pixels, e.g., greenpixel G at location 501 in row 510 (FIG. 5B), process 302 generates avertical estimated red pixel r_v and a horizontal estimated blue pixelb_h. For each green pixel in a row of red and green pixels, e.g., agreen pixel G at location 502 is in row 511, process 302 generates ahorizontal estimated red pixel r_h and a vertical estimated blue pixelb_v.

For each red pixel, e.g., a red pixel R at location 503, process 302generates a vertical estimated green pixel g_v and a horizontalestimated green pixel g_h. For each blue pixel, e.g., blue pixel B atlocation 504, process 302 generates a vertical estimated green pixel g_vand a horizontal estimated green pixel g_h.

Thus, following completion of process 302, each location in block 500has the information illustrated in FIG. 5B. In the drawings, a locationin block 500 is bounded by a square. Following completion of process302, processing transfers to GENERATE COLOR DIFFERENCES process 303,sometimes referred to as process 303.

Process 303 generates color difference signals, i.e., green-red colordifference signals and green-blue color difference signals. For a greenpixel G in a row of green and blue pixels, a horizontal green-blue colordifference signal GBdiff_h(x, y) is defined as:

GBdiff_h(x, y) = G(x, y) − b_h(x, y)

For a green pixel G in a column of green and blue pixels, a verticalgreen-blue color difference signal GBdiff_v(x, y) is defined as:

GBdiff_v(x, y) = G(x, y) − b_v(x, y)

When a horizontal estimated green pixel g_h(x, y) that is estimated froma row of green pixels is used, horizontal green-blue color differencesignal GBdiff_h(x, y) for a blue pixel B is defined as:

GBdiff_h(x, y) = g_h(x, y) − B(x, y)

When a vertical estimated green pixel g_v(x, y) that is estimated from acolumn of green pixels is used, vertical green-blue color differencesignal GBdiff_v(x, y) for a blue pixel B, is defined as:

GBdiff_v(x, y) = g_v(x, y) − B(x, y)

For a green pixel G in a row of green and red pixels, a horizontalgreen-red color difference signal GRdiff_h(x, y) is defined as:

GRdiff_h(x, y) = G(x, y) − r_h(x, y)

For a green pixel G in a column of green and red pixels, a verticalgreen-red color difference signal GRdiff_v(x, y) is defined as:

GRdiff_v(x, y) = G(x, y) − r_v(x, y)

When a horizontal estimated green pixel g_h(x, y) that is estimated froma row of green pixels is used, horizontal green-red color differencesignal GRdiff_h(x, y) for a red pixel R is defined as:

GRdiff_h(x, y) = g_h(x, y) − R(x, y)

When a vertical estimated green pixel g_v(x, y) that is estimated from acolumn of green pixels is used, vertical green-red color differencesignal GRdiff_v(x, y) for a red pixel R is defined as:

GRdiff_v(x, y) = g_v(x, y) − R(x, y)

Thus, for green pixel G at location 501 in row 510 of green and bluepixels, process 303 generates horizontal green-blue color differencesignal GBdiff_h(x, y) and vertical green-red color difference signalGRdiff_v(x, y). For blue pixel B at location 504 in row 510 of green andblue pixels, process 303 generates horizontal green-blue colordifference signal GBdiff_h(x, y) and vertical green-blue colordifference signal GBdiff_v(x, y).

For green pixel G at location 502 in row 511 of green and red pixels,process 303 generates horizontal green-red color difference signalGRdiff_h(x, y) and vertical green-blue color difference signalGBdiff_v(x, y). For red pixel R at location 503 in row 511 of green andred pixels, process 303 generates horizontal green-red color differencesignal GRdiff_h(x, y) and vertical green-red color difference signalGRdiff_v(x, y).

Following completion of process of 303 the location of each green pixelG has one of a horizontal green-blue color difference signal GBdiff_h(x,y) and a horizontal green-red color difference signal GRdiff_h(x, y) andone of a vertical green-blue color difference signal GBdiff_v(x, y) anda vertical green-red color difference signal GRdiff_v(x, y). Thelocation of each blue pixel B has a horizontal green-blue colordifference signal GBdiff_h(x, y) and a vertical green-blue colordifference signal GBdiff_v(x, y). The location of each red pixel R has ahorizontal green-red color difference signal GRdiff_h(x, y) and verticalgreen-red color difference signal GRdiff_v(x, y). FIG. 5C shows thecolor difference signals for the pixels and estimated pixels in FIG. 5B.

After the color difference signals are available, FILTER COLORDIFFERENCES process 304, sometimes referred to as process 304, low passfilters the color differences at each location using low pass filter 600(FIG. 6). In one aspect, a nine-tap Gaussian filter is used.

Horizontal green-red color difference signal GRdiff_h(x, y) is filteredby horizontal low pass filter 601 to generate filtered horizontalgreen-red color difference signal G.Rdiff_h(x, y). Vertical green-redcolor difference signal GRdiff_v(x, y) is filtered by vertical low passfilter 602 to generate filtered vertical green-red color differencesignal G.Rdiff_v(x, y).

Horizontal green-blue color difference signal GBdiff_h(x, y) is filteredby horizontal low pass filter 603 to generate filtered horizontalgreen-blue color difference signal G.Bdiff_h(x, y). Vertical green-bluecolor difference signal GBdiff_v(x, y) is filtered by vertical low passfilter 604 to generate filtered and vertical green-blue color differencesignal G.Bdiff_v(x, y).

The use of four low pass filters in FIG. 6 is illustrative only and isnot intended to be limiting. The low pass filtering can be implementedwith more or less than four low pass filters. The number of filters useddepends on the real-time requirements and cost. For example, a singlelow pass filter could be used, or four low pass filters could be used.

FIG. 7 illustrates the filtered signals at each location in block 500.Following completion of FILTER COLOR DIFFERENCES process 304 processingtransfers to ADAPTIVE RECONSTRUCTION OF GREEN PIXELS process 305,sometimes referred to as process 305.

FIG. 8 is a process flow diagram of process 305, in one aspect. The useof a linear process flow is illustrative only and is not intended to belimiting. In view of the following description, process 305 can beimplemented in anyway that achieves each of the acts described. Forexample, process 305 could be implemented using two hardwarepipelines—one pipeline that processes red pixels and a second pipelinethat processes blue pixels. The red pixel pipeline generates areconstructed green pixel g for each red pixel R, and the blue pixelpipeline generates a reconstructed green pixel g for each blue pixel B.

Alternatively, a single pipeline could be used. Examination of FIG. 8shows that the same processes are used in the two pipelines and so couldbe implemented as a single pipeline. Herein, two pipelines, i.e.,processes 811 to 816 and process 821 to 826, are used to assist invisualizing the processing of each captured pixel. However, processes811 and 821 are equivalent. Check processes 812 and 822 are equivalent.Processes 813 and 823 are equivalent. Check processes 814 and 824 areequivalent. Processes 815 and 825 are equivalent, and processes 816 and826 are equivalent.

Initially, RED PIXEL check process 810 determines whether the pixel atcurrent location (x, y) is red pixel R. If the pixel is a red pixel R,RED PIXEL check process 810 transfer to GENERATE LOCAL EDGE ESTIMATORSprocess 811, sometimes referred to as process 811, and otherwise to BLUEPIXEL check process 820.

GENERATE LOCAL EDGE ESTIMATORS process 811 generates a vertical localedge estimator dV (of absolute value) and a horizontal local edgeestimator dH (of absolute value), as described more completely below.Following generation of the local edge estimators, GENERATE LOCAL EDGEESTIMATORS process 811 transfers to a first check process 812.

First check process 812 determines whether horizontal local estimator dHminus a robust factor Δ is larger than vertical local edge estimator dV.Stated in another way, check process 812 determines whether horizontallocal estimator dH minus vertical local edge estimator dH is larger thanrobust factor Δ. If horizontal local estimator dH minus vertical localedge estimator dV is larger than robust factor Δ, there is a horizontaledge. Thus, check process 812 transfers to GENERATE GREEN PIXEL WITHFILTERED HORIZONTAL GREEN-RED DIFFERENCE process 813, sometimes referredto as process 813, and otherwise transfers to second check process 814.The usage of a small value robust factor Δ is chosen to preventdetection of false edges or a sudden shift, for example, from a verticaledge to horizontal edge, enabling a more robust process. In one aspect,robust factor Δ is normalized against the absolute pixel value. Forexample, robust factor Δ should be scaled up proportionally if the pixelgoes from an eight-bit value to a ten-bit value.

In one aspect, the value of robust factor Δ is empirically determinedbased, for example, on noise levels and expected sharpness of edges inthe captured images. For example, the local edge estimators at severaltypical locations—vertical edge, horizontal edge or no edge—wereexamined. In summary, the value of robust factor Δ that is selecteddepends on the image gradient values that are used in the edgeestimators, hence the absolute value of a pixel (say 256/8 bit), howsharp an edge could be expected in an image, and noise level (largernoise implies larger delta) for typical images that are captured.

When check process 812 transfers to GENERATE GREEN PIXEL WITH FILTEREDHORIZONTAL GREEN-RED DIFFERENCE process 813, the edge effect at location(x, y) is greater in the horizontal direction than in the verticaldirection. Thus, the filtered horizontal green-red color differencesignal provides a better estimate of the characteristics of green pixelg at location (x, y). GENERATE GREEN PIXEL WITH FILTERED HORIZONTALGREEN-RED DIFFERENCE process 813 generates a reconstructed green pixel gat location (x, y) using the filtered horizontal green-red colordifference signal G.Rdiff_h at location (x, y). Specifically, in oneaspect:

g(x, y) = R(x, y) + G.Rdiff_h(x, y)

Following generating reconstructed green pixel g(x, y), GENERATE GREENPIXEL WITH FILTERED HORIZONTAL GREEN-RED DIFFERENCE process 813transfers to LAST PIXEL check process 830.

If first check process 812 transfers to second check process 814, secondcheck process 814 determines whether vertical local estimator dV minusrobust factor Δ is larger than horizontal local edge estimator dH.Stated in another way, check process 814 determines whether verticallocal estimator dV minus horizontal local edge estimator dH is largerthan robust factor Δ. If vertical local estimator dV minus horizontallocal edge estimator dH is larger than robust factor Δ, there is avertical edge. Thus, check process 814 transfers to GENERATE GREEN PIXELWITH FILTERED VERTICAL GREEN-RED DIFFERENCE process 815, sometimesreferred to as process 815, and otherwise transfers to GENERATE GREENPIXEL WITH AVERAGE OF FILTERED VERTICAL GREEN-RED DIFFERENCE ANDFILTERED HORIZONTAL GREEN-RED DIFFERENCE process 816, sometimes referredto as process 816.

When check process 814 transfers to GENERATE GREEN PIXEL WITH FILTEREDVERTICAL GREEN-RED DIFFERENCE process 815, the edge effect is greater inthe vertical direction than in the horizontal direction at location (x,y). Thus, the filtered vertical green-red color difference signalprovides a better estimate of the characteristics of green pixel g atlocation (x, y). GENERATE GREEN PIXEL WITH FILTERED VERTICAL GREEN-REDDIFFERENCE process 815 generates a reconstructed green pixel g atlocation (x, y) using the filtered vertical green-red color differencesignal G.Rdiff_v at location (x, y). Specifically, in one aspect:

g(x, y) = R(x, y) + G.Rdiff_v(x, y)

Following generating reconstructed green pixel g(x, y), GENERATE GREENPIXEL WITH FILTERED VERTICAL GREEN-RED DIFFERENCE process 815 transfersto LAST PIXEL check process 830.

When check process 814 transfers to GENERATE GREEN PIXEL WITH AVERAGE OFFILTERED VERTICAL GREEN-RED DIFFERENCE AND FILTERED HORIZONTAL GREEN-REDDIFFERENCE process 816, the edge effects in the horizontal direction andvertical direction at location (x, y) are about equal, e.g., aredifferent by an amount less than robust factor Δ. Thus, GENERATE GREENPIXEL WITH AVERAGE OF FILTERED VERTICAL GREEN-RED DIFFERENCE ANDFILTERED HORIZONTAL GREEN-RED DIFFERENCE process 816 generates areconstructed green pixel g at location (x, y) using the average offiltered vertical green-red color difference signal G.Rdiff_v atlocation (x, y) and filtered horizontal green-red color differencesignal G.Rdiff_h at location (x, y). Specifically, in one aspect:

g(x, y) = R(x, y) + ((G.Rdiff_v(x, y) + G.Rdiff_h(x, y))/2)

Following generating reconstructed green pixel g(x, y), GENERATE GREENPIXEL WITH AVERAGE OF FILTERED VERTICAL GREEN-RED DIFFERENCE ANDFILTERED HORIZONTAL GREEN-RED DIFFERENCE process 816 transfers to LASTPIXEL check process 830.

When RED PIXEL check process 810 transfers to BLUE PIXEL check process820, BLUE PIXEL check process 820 determines whether the pixel atcurrent location (x, y) is a blue pixel B. If the pixel is a blue pixelB, BLUE PIXEL check process 820 transfer to GENERATE LOCAL EDGEESTIMATORS process 821, sometimes referred to as process 821, andotherwise to LAST PIXEL check process 830.

GENERATE LOCAL EDGE ESTIMATORS process 821 generates vertical local edgeestimator dV and horizontal local edge estimator dH, as described morecompletely below. Following generation of the local edge estimators,GENERATE LOCAL EDGE ESTIMATORS process 821 transfers to a third checkprocess 822.

Third check process 822 determines whether horizontal local estimator dHminus robust factor Δ is larger than vertical local edge estimator dV.Stated in another way, check process 822 determines whether horizontallocal estimator dH minus vertical local edge estimator dV is larger thanrobust factor Δ. If horizontal local estimator dH minus vertical localedge estimator dV is larger than robust factor Δ, there is a horizontaledge. Thus, check process 822 transfers to GENERATE GREEN PIXEL WITHFILTERED HORIZONTAL GREEN-BLUE DIFFERENCE process 823, sometimesreferred to as process 823, and otherwise transfers to fourth checkprocess 824.

When check process 822 transfers to GENERATE GREEN PIXEL WITH FILTEREDHORIZONTAL GREEN-BLUE DIFFERENCE process 823, the edge effect atlocation (x, y) is greater in the horizontal direction than in thevertical direction. The filtered horizontal green-blue color differencesignal provides a better estimate of the characteristics of green pixelg at location (x, y). Thus, GENERATE GREEN PIXEL WITH FILTEREDHORIZONTAL GREEN-BLUE DIFFERENCE process 823 generates a reconstructedgreen pixel g at location (x, y) using the filtered horizontalgreen-blue color difference signal G.Bdiff_h at location (x, y).Specifically, in one aspect:

g(x, y) = B(x, y) + G.Bdiff_h(x, y)

Following generating reconstructed green pixel g(x, y), GENERATE GREENPIXEL WITH FILTERED HORIZONTAL GREEN-BLUE DIFFERENCE process 823transfers to LAST PIXEL check process 830.

If third check process 822 transfers to fourth check process 824, fourthcheck process 824 determines whether vertical local estimator dV minusrobust factor Δ is larger than horizontal local edge estimator dH.Stated in another way, check process 824 determines whether verticallocal estimator dV minus horizontal local edge estimator dH is largerthan robust factor Δ. If vertical local estimator dV minus horizontallocal edge estimator dH is larger than robust factor Δ, there is avertical edge. Thus, check process 824 transfers to GENERATE GREEN PIXELWITH FILTERED VERTICAL GREEN-BLUE DIFFERENCE process 825, sometimesreferred to as process 825, and otherwise transfers to GENERATE GREENPIXEL WITH AVERAGE OF FILTERED VERTICAL GREEN-BLUE DIFFERENCE ANDFILTERED HORIZONTAL GREEN-BLUE DIFFERENCE process 826, sometimesreferred to as process 826.

When check process 824 transfers to GENERATE GREEN PIXEL WITH FILTEREDVERTICAL GREEN-BLUE DIFFERENCE process 825, the edge effect is greaterin the vertical direction than in the horizontal direction at location(x, y). Thus, the filtered vertical green-blue color difference signalprovides a better estimate of the characteristics of green pixel g atlocation (x, y). GENERATE GREEN PIXEL WITH FILTERED VERTICAL GREEN-BLUEDIFFERENCE process 825 generates a reconstructed green pixel g atlocation (x, y) using the filtered vertical green-blue color differencesignal G.Bdiff_v at location (x, y). Specifically, in one aspect:

g(x, y) = B(x, y) + G.Bdiff_v(x, y)

Following generating reconstructed green pixel g(x, y), GENERATE GREENPIXEL WITH FILTERED VERTICAL GREEN-BLUE DIFFERENCE process 825 transfersto LAST PIXEL check process 830.

When check process 824 transfers to GENERATE GREEN PIXEL WITH AVERAGE OFFILTERED VERTICAL GREEN-BLUE DIFFERENCE AND FILTERED HORIZONTALGREEN-BLUE DIFFERENCE process 826, the edge effects in the horizontaldirection and vertical direction at location (x, y) are about equal,e.g., a different by an amount less than robust factor Δ. Thus, GENERATEGREEN PIXEL WITH AVERAGE OF FILTERED VERTICAL GREEN-BLUE DIFFERENCE ANDFILTERED HORIZONTAL GREEN-BLUE DIFFERENCE process 826 generates areconstructed green pixel g at location (x, y) using the average offiltered vertical green-blue color difference signal G.Bdiff_v atlocation (x, y) and filtered horizontal green-blue color differencesignal G.Bdiff_h at location (x, y).

g(x, y) = B(x, y) + ((G.Bdiff_v(x, y) + G.Bdiff_h(x, y))/2)

Following generating reconstructed green pixel g(x, y), GENERATE GREENPIXEL WITH AVERAGE OF FILTERED VERTICAL GREEN-BLUE DIFFERENCE ANDFILTERED HORIZONTAL GREEN-BLUE DIFFERENCE process 826 transfers to LASTPIXEL check process 830.

When processing transfers to LAST PIXEL check process 830, check process830 determines whether all the pixels in the frame have been processed.If all the pixels have been processed, check process 830 transfers toRECONSTRUCT RED AND BLUE PIXELS process 311 and otherwise transfers toNEXT PIXEL process 831. NEXT PIXEL process 831 moves the current pixelpointer from the current pixel to the next pixel in the frame and thentransfers to RED PIXEL check process 810. Typically, this sequentialprocess of GREEN first, and then RED and BLUE is not done sequentiallypixel by pixel for the complete frame. It is a sequential process withina selected portion of the frame, e.g., a block of the frame, forreal-time processing. In one aspect, this allows multiple blocks in aframe to be sequentially processed in parallel and so the processing canbe done in real-time. The parallel processing of blocks within a frameis also applicable to the other processes described below.

When check process 830 transfers to RECONSTRUCT RED AND BLUE PIXELSprocess 311, each location in the frame has either a green pixel G or areconstructed green pixel g. FIG. 9 shows information available in pixelblock 500 following completion of ADAPTIVE RECONSTRUCTION OF GREENPIXELS process 305.

RECONSTRUCT RED AND BLUE PIXELS process 311 can be implemented in avariety of ways. The generation of the missing green pixels is mostcritical and so the process has higher spatial resolution. The detailedspatial resolution is typically not required for the missing red andblue pixels and so a simpler reconstruction process can be used. FIG. 10is a more detailed process flow diagram of one implementation ofRECONSTRUCT RED AND BLUE PIXELS process 311. This diagram isillustrative only and is not intended to be limiting. The illustratedprocess flow and the order of the process flow is used for ease ofdiscussion and is not intended to limit process 311 to either thespecific acts or the specific sequence of acts shown. In view of thisdisclosure, one knowledgeable in the field can implement process 311taking into consideration processor capability, available memory, timingrequirements, etc.

In RECONSTRUCT RED AND BLUE PIXELS process 311, a reconstructed bluepixel b(x, y) is generated for each red pixel R(x, y), and then areconstructed red pixel r(x, y) is generated for each blue pixel B(x,y). As described more completely below, each of these reconstructionsuse a sum of reconstructed green pixel g(x, y) and the average of colordifference signals at the four diagonal neighboring pixels.

Following these reconstructions, each red pixel R(x, y) has areconstructed green pixel and a reconstructed blue pixel. Each bluepixel B(x, y) has a reconstructed red pixel r(x, y) and a reconstructedgreen pixel g(x, y). Next, a reconstructed red pixel and r(x, y) and areconstructed blue pixel b(x, y) are generated for each green pixel G(x,y).

Specifically, for the example of RECONSTRUCT RED AND BLUE PIXELS process311 in FIG. 10, processing transfers from RECONSTRUCT GREEN PIXELSprocess 310 to RED PIXEL check process 1010. RED PIXEL check process1010 determines whether the current pixel at location (x, y) is a redpixel R(x, y). If the current pixel at location (x, y) is a red pixelR(x, y), RED PIXEL check process 1010 transfers to GENERATE BLUE PIXELprocess 1011, and otherwise transfers to BLUE PIXEL check process 1012.

GENERATE BLUE PIXEL process 1011 generates a reconstructed blue pixelb(x,y). Specifically, for a red pixel R(x, y), see FIG. 11A,reconstructed blue pixel b(x, y) is generated as a sum of reconstructedgreen pixel g(x, y) and the average of color difference signals (betweenthe original blue pixel and the reconstructed green pixel) at the fourdiagonal neighboring pixels, i.e.,

b(x, y) = g(x, y) − ΔGB_R(x, y)whereΔGB_R(x, y)= − [(B(x − 1, y + 1) − g(x − 1, y + 1)) + (B(x − 1, y − 1) − g(x − 1, y − 1)) + (B(x + 1, y − 1) − g(x + 1, y − 1)) + (B(x + 1, y + 1) − g(x + 1, y + 1))]/4

Following generation of reconstructed blue pixel b(x, y), GENERATE BLUEPIXEL process 1011 transfers to LAST PIXEL check process 1014.

When RED PIXEL check process 1010 transfers to BLUE PIXEL check process1012, check process 1012 determines whether the current pixel atlocation (x, y) is a blue pixel B(x, y). If the current pixel atlocation (x, y) is a blue pixel B(x, y), BLUE PIXEL check process 1012transfers to GENERATE RED PIXEL process 1013, and otherwise transfers toLAST PIXEL check process 1014.

GENERATE RED PIXEL process 1013 generates a reconstructed red pixelr(x,y). Specifically, for an original blue pixel B(x, y), see FIG. 11B,a reconstructed red pixel r(x, y) is generated as the sum ofreconstructed green pixel g(x, y) and the average of color differencesignals (between the original red pixel and the reconstructed greenpixel) at the four diagonal neighboring pixels, i.e.,

r(x, y) = g(x, y) − ΔGR_B(x, y)whereΔGR_B(x, y)= − [(R(x − 1, y + 1) − g(x − 1, y + 1)) + (R(x − 1, y − 1) − g(x − 1, y − 1)) + (R(x + 1, y − 1) − g(x + 1, y − 1)) + (R(x + 1, y + 1) − g(x + 1, y + 1))]/4

Following generation of reconstructed red pixel r(x, y), GENERATE REDPIXEL process 1013 transfers to LAST PIXEL check process 1014.

LAST PIXEL check process 1014 determines whether all the pixels in theframe have been processed. If all the pixels have been processed, LASTPIXEL check process 1014 resets the current pixel pointer to the startof the frame and then transfers to GREEN PIXEL check process 1015.Conversely, if all the pixels have not been processed, LAST PIXEL checkprocess 1014 adjusts a current pixel pointer to the next pixel and thentransfers to RED PIXEL check process 1010.

When LAST PIXEL check process 1014 transfers to GREEN PIXEL checkprocess 1015, all reconstructed red pixels r(x, y) on original pixelsB(x, y) and all reconstructed blue pixels b(x, y) original on red pixelsR(x, y) have been generated, and so the reconstructed red pixels r(x, y)and the reconstructed blue pixels b(x, y) on original green pixels G(x,y) can be generated.

Thus, GREEN PIXEL check process 1015 determines whether the currentpixel at location (x, y) is a green pixel G(x, y). If the current pixelat location (x, y) is a green pixel G(x, y), GREEN PIXEL check process1015 transfers to BLUE-GREEN ROW check process 1016, and otherwisetransfers to LAST PIXEL check process 1021.

The processing that is performed to generate the reconstructed redpixels r(x, y) and the reconstructed blue pixels b(x, y) on originalgreen pixels G(x,y) depends on the color of the pixels surroundingoriginal green pixel G(x, y). Thus, BLUE-GREEN ROW check process 1016determines whether original green pixel G(x, y) is in a row of green andblue pixels. If original green pixel G(x, y) is in a row of green andblue pixels, check process 1016 transfers to a first GENERATE BLUE PIXELprocess 1017 and otherwise transfers to a second GENERATE BLUE PIXELprocess 1019.

When BLUE-GREEN ROW check process 1016 transfers to first GENERATE BLUEPIXEL process 1017, process 1017 generates a reconstructed blue pixelb(x,y) at location (x, y) of original green pixel G(x, y). Specifically,for original green pixel G(x, y), see FIG. 11C, a reconstructed bluepixel b(x, y) is generated as the sum of original green pixel G(x, y)and the average of color difference signals (between one of the originalblue pixel or the reconstructed blue pixel from process 1011 and thereconstructed green pixel) at the four neighboring pixels, i.e.,

b(x, y) = G(x, y) − ΔGB_G(x, y)whereΔGB_G(x, y) = −([(B(x − 1, y) − g(x − 1, y)) + (B(x + 1, y) − g(x + 1, y)) + (b(x, y − 1) − g(x, y − 1)) + (b(x, y + 1) − g(x, y + 1))]/4

Upon completion, first GENERATE BLUE PIXEL process 1017 transfersprocessing to a first GENERATE RED PIXEL process 1018.

In first GENERATE RED PIXEL process 1018, a reconstructed red pixel r(x,y) is generated at location (x, y) of original green pixel G(x, y).Specifically, for original green pixel G(x, y), see FIG. 11C, areconstructed red pixel r(x, y) is generated as the sum of originalgreen pixel G(x, y) and the average of color difference signals (betweenone of the reconstructed red pixel from process 1013 or the original redpixel and the reconstructed green pixel) at the four neighboring pixels,i.e.,

r(x, y) = G(x, y) − ΔGR_G(x, y)whereΔGR_G(x, y) = −([(r(x − 1, y) − g(x − 1, y)) + (r(x + 1, y) − g(x + 1, y)) + (R(x, y − 1) − g(x, y − 1)) + (R(x, y + 1) − g(x, y + 1))]/4

Upon completion, first GENERATE RED PIXEL process 1018 transfersprocessing to LAST PIXEL check process 1021.

When BLUE-GREEN ROW check process 1016 transfers to second GENERATE BLUEPIXEL process 1019, process 1019 generates a reconstructed blue pixelb(x,y) at location (x, y) of original green pixel G(x, y). Specifically,for original green pixel G(x, y), see FIG. 11D, a reconstructed bluepixel b(x, y) is generated as the sum of original green pixel G(x, y)and the average of color difference signals (between one of the originalblue pixel or the reconstructed blue pixel from process 1011 and thereconstructed green pixel) at the four neighboring pixels, i.e.,

b(x, y) = G(x, y) − ΔGB_G(x, y)whereΔGB_G(x, y) = −([(b(x − 1, y) − g(x − 1, y)) + (b(x + 1, y) − g(x + 1, y)) + (B(x, y − 1) − g(x, y − 1)) + (B(x, y + 1) − g(x, y + 1))]/4

Upon completion, second GENERATE BLUE PIXEL process 1019 transfersprocessing to a second GENERATE RED PIXEL process 1020.

In second GENERATE RED PIXEL process 1020, a reconstructed red pixelr(x, y) is generated at location (x, y) of original green pixel G(x, y).Specifically, for original green pixel G(x, y), see FIG. 11D, areconstructed red pixel r(x, y) is generated as the sum of originalgreen pixel G(x, y) and the average of color difference signals (betweenone of the reconstructed red pixel from process 1013 or the original redpixel and the reconstructed green pixel) at the four neighboring pixels,i.e.,

r(x, y) = G(x, y) − ΔGR_G(x, y)whereΔGR_G(x, y) = −([(R(x − 1, y) − g(x − 1, y)) + (R(x + 1, y) − g(x + 1, y)) + (r(x, y − 1) − g(x, y − 1)) + (r(x, y + 1) − g(x, y + 1))]/4

Upon completion, second GENERATE RED PIXEL process 1020 transfersprocessing to LAST PIXEL check process 1021.

LAST PIXEL check process 1021 determines whether all the pixels in theframe have been processed. If all the pixels have been processed, LASTPIXEL check process transfers to SEND FULL RESOLUTION FRAME process 312.Conversely, if all the pixels have not been processed, LAST PIXEL checkprocess 1021 adjusts a current pixel pointer to the next pixel and thentransfers to GREEN PIXEL check process 1015. When LAST PIXEL checkprocess 1021 transfers to SEND FULL RESOLUTION FRAME process 312, eachlocation in the frame has a complete set of red, green, and blue pixels.

As described above, GENERATE ESTIMATE OF MISSING PIXELS process 302 usesa second order series expansion in generating reconstructed pixels at alocation in block 500. For example, when the current pixel is a greenpixel G at location (x, y) (FIG. 12A or 12B), process 302 uses thesecond order series expansion to generate estimated values for the redand blue pixels at location (x, y). When the current pixel is a redpixel R at location (x, y) (FIG. 12C), process 302 uses the second orderseries expansion to generate estimated horizontal and vertical valuesfor the green pixel at location (x, y). When the current pixel is a bluepixel B is at location (x, y) (FIG. 12D), process 302 uses the secondorder series expansion to generate horizontal estimated and verticalestimated pixel values for a green pixel at location (x, y).

The value of the pixel at location (x, y) is represented by a functionf(x, y). A second order Taylor series expansion about x can be expressedas:

f(x + 1, y) = f(x, y) + fx^(′)(x, y) + fx^(″)(x, y)/2andf(x − 1, y) = f(x, y) − fx^(′)(x, y) + fx^(″)(x, y)/2.

where the first order gradient fx′(x, y) can be approximated as:

fx^(′)(x, y) = [f(x + 1, y) − f(x − 1, y)]/2,

andwhere the second order gradient fx″(x, y) can be approximated as:

fx^(″)(x, y) = [f(x + 2, y) + f(x − 2, y) − 2 * f(x, y)]/4.

Through simple manipulations, it can be shown that the interpolationsf_h(x, y) and f_v(x, y) for missing pixels can be expressed as follows(using the second order Taylor series expansion)

Horizontally along x:

f_h(x, y) = [f(x + 1, y) + f(x − 1, y)]/2 −  [f(x + 2, y) + f(x − 2, y) − 2 * f(x, y)]/4

Vertically along y:

f_v(x, y) = [f(x, y + 1) + f(x, y − 1)]/2 −  [f(x, y + 2) + f(x, y − 2) − 2 * f(x, y)]/4

The expressions are used in one aspect to generate vertical estimatedred pixel r_v, horizontal estimated red pixel r_h, vertical estimatedgreen pixel g_v, horizontal estimated green pixel g_h, verticalestimated blue pixel b_v, and vertical estimated blue pixel b_v inprocess 302.

These interpolations make sense based on the assumption that the secondorder gradients of different components are similar. For example, iflocation (x, y) is a green pixel (FIG. 12A or 12B) in a Bayer patterncolor filter array, locations (x−1, y) and (x+1, y) are either red orblue pixels and (x−2, y) and (x+2, y) are green pixels. As a result, theinterpolation would be good for either a missing red pixel or a missingblue pixel at location (x, y). When the pixel at location (x, y) is ared pixel (FIG. 12C), locations (x−1, y) and (x+1, y) are Green pixelsand locations (x−2, y) and (x+2, y) are red pixels. As a result, theinterpolation is good for a missing green pixel at location (x, y). Theconcepts in this paragraph and the previous paragraph are known to thoseknowledgeable in the field.

As described above, in ADAPTIVE RECONSTRUCTION OF GREEN PIXELS process305, GENERATE LOCAL EDGE ESTIMATORS process 811 generates vertical localedge estimator dV and horizontal local edge estimator dH. The localestimators can be either a first order gradient or a combination of afirst order gradient and a second order gradient.

For location (x, y), when horizontal local edge estimator dH is a firstorder gradient obtained using the above Taylor series expansion,horizontal local edge estimator dH is:

dH = ❘fx^(′)(x, y)❘ = ❘[f(x + 1, y) − f(x − 1, y)]/2❘

For location (x, y), when vertical local edge estimator dV is a firstorder gradient obtained using the above Taylor series expansion,vertical local edge estimator dV is:

dV = ❘fy^(′)(x, y)❘ = ❘[f(x, y + 1) + f(x, y − 1)]/2❘

For location (x, y), when horizontal local edge estimator dH is sum ofthe first order gradient obtained using the above Taylor seriesexpansion and the above estimation of the second order gradient,horizontal local edge estimator dH is:

dH = ❘fx^(′)(x, y)❘ + 2 * ❘fx^(″)(x, y)❘dH = ❘[f(x + 1, y) + f(x − 1, y)]/2❘+❘[f(x + 2, y) + f(x − 2, y) − 2 * f(x, y)]/2❘

For location (x, y), when vertical local edge estimator dV is sum of thefirst order gradient obtained using the above Taylor series expansionand the above estimation of the second order gradient, vertical localedge estimator dV is:

dV = ❘fy^(′)(x, y)❘ + 2 * ❘fy^(″)(x, y)❘dV = ❘[f(x, y + 1) + f(x, y − 1)]/2❘ + ❘[f(x, y + 2) + f(x, y − 2) − 2f(x, y)]/2❘

In another aspect, local contrast enhancement is implemented in theenhanced demosaicing process. Human perception of a color image issensitive to the local brightness change, i.e., local contrast, of acolor image. Thus, the demosaicing process with local contrastenhancement is designed to increase the local contrast while preservingthe apparent global brightness to the extent possible, via discardingsome or all of one or more color components of an original color imagewhen determining a new brightness of pixels in a new contrast enhancedcolor image.

The efficient demosaicing process with local contrast enhancement, whichis described more completely below, digitally enhances local contrastfor surgical images based on physical properties of imaging and humanperception of surgical image content. The contrast enhanced color imagesallow surgeons to more easily differentiate critical structures such asblood vessels from surrounding tissues.

In surgical imaging, the predominant color is red for both tissues andblood vessels, making it difficult for surgeons to differentiatecritical structures from surrounding tissues. In addition, the red colorcomponent appears to be fuzzy compared to the green and blue colorcomponents, apparently due to relatively deeper tissue penetration ofcomplex biological structures by the red color component. This isespecially true in close-range imaging. This makes it even moredifficult to differentiate fine structures from surrounding tissues. Theefficient demosaicing process with local contrast enhancement describedbelow enhances the color image contrast by removing, or at leastreducing, the contrast contribution from the red and blue colorcomponents while keeping the true color of the color images unchanged.

In the following example, color images of surgical sites are considered.However, the use of surgical images is illustrative only and is notintended to be limiting. The methods described below for enhancement ofcontrast are also applicable to a system in which the response for oneof a plurality of components in a color space is degraded. Thedegradation could be the result of poor optical system response, poorcalibration of electronic components, failure of electronic components,etc. In each instance, the component in the color space associated withthe degradation is not used in determining the brightness that providesenhanced contrast. For a more detailed description of contrastenhancement, see U.S. Patent Publication No. US 2012/0209287 A1(disclosing “Method and Structure for Image Local ContrastEnhancement”), which is incorporated by reference.

The color components of the full resolution frame created in theenhanced demosaicing process are defined by a color space. Typical colorspaces include: a color space that includes a red color component R, agreen color component G, and a blue color component B, which is referredto as a RGB color space; a YUV color space that includes a luminancecomponent Y and two chromatic components U and V; a YCbCr color spacethat includes a luminance component Y and two chromatic components Cband Cr; and a color space that includes a hue component, a saturationcomponent, and a brightness component. This list of color spaces and ofthe components included in the color spaces is illustrative only and isnot intended to be limiting to these specific color spaces. Aspects ofthe invention may be applied in various color spaces.

The luminance components in the YUV and YCbCr color spaces and thebrightness component in the hue, saturation, and brightness color spacerepresent the apparent brightness of a pixel. The apparent brightness isdefined by a grayscale. The grayscale has a range from black to whitewith varying shades of grey between black and white. The values of thegrayscale, in one aspect, range from 0 to 255, where a value of zerorepresents black and a value of 255 represents white. Values betweenzero and 255 represent various shades of gray.

Herein, a brightness component refers to a component in a color spacethat is an apparent brightness and that is a grayscale component. Thebrightness component is not a color component because the brightnesscomponent conveys only brightness information and not color information.In color spaces with a brightness component, the color components arethe components of the color space other than the brightness componentand are referred to herein as chromatic components.

In the following example the YCbCr color space is used. However, this isillustrative only and is not intended to be limiting. The processdescribed is applicable to any color space that includes a brightnesscomponent.

FIG. 13 is a process flow diagram of one aspect of ENHANCED DEMOSAICINGprocess 1300 with local contrast enhancement, sometimes referred to asprocess 1300. In process 1300, RECEIVE CAPTURED FRAME process 301 andRECONSTRUCT GREEN PIXELS process 310 are the same as described abovewith respect to FIG. 3 and so that description is not repeated forprocess 1300, but that description is incorporated by reference.Following completion of process 310, the green pixels have beenreconstructed and processing transfers to ENHANCE BRIGHTNESS PIXEL INSECOND COLOR SPACE process 1320, sometimes referred to as process 1320.

Prior to considering process 1320, it is informative to consider anormal transformation from the RGB color space to the YCbCr color space.The transformation from the RGB color space to the YCbCr color space fora pixel is known. The transformation starts with a basic transform,which can be represented as:

$\begin{matrix}{Y = {{16} + {t11*R} + {t12*G} + {t13*B}}} & ( {1a} ) \\{{Cb} = {128 + {t21*R} + {t22*G} + {t23*B}}} & ( {1b} ) \\{{Cr} = {128 + {t31*R} + {t32*G} + {t33*B}}} & ( {1c} )\end{matrix}$

For example, t11=65.481, t12=128.553, t13=24.966; t21=−37.797,t22=−74.203, t23=112; and t31=112, t32=−93.786, t33=−18.214;

Definition (1a) is an example of a standards transformation that mapsall color components of a first color space into a brightness componentof a second color space. This transformation is defined by industrystandards and the values of coefficients t11, t12, and t13 are definedby those industry standards. Therefore, definition (1a) is referred toas a standards transformation of color components in a first color spaceto a brightness component in a second color space.

To implement definition (1a) requires the red pixel and the blue pixelat each location, but the red and blue pixels at some locations are notavailable to process 1320. Thus, in process 1320, brightness componentYnew is defined as:

$\begin{matrix}{{Ynew} = {16 + {S*t12*G}}} & ( {2a} )\end{matrix}$

where S is a scale factor. The contributions of the red and blue pixelsto the brightness component are not used and the contribution of thegreen pixel to the brightness is scaled to compensate for the loss ofthe contribution of the red and blue pixels. Process 1320 usesdefinition (2a) to reconstruct a brightness pixel for each location inthe frame. Following completion of process 1320, processing transfers toGENERATE CHROMINANCE PIXELS IN SECOND COLOR SPACE process 1321.

Scale factor S is empirically determined. For example, a RGB image istransformed to an image in the YCbCr color space using definitions (2a),(1b), and (1c). The transformation is done using a plurality of scalefactors, e.g., 1.3, 1.4, and 1.5. The resulting images are reviewed by agroup of individuals and the value of the scale factor for the imageperceived as having the best overall contrast is selected as the valueof scale factor S. In one aspect a value of 1.4 was used for scalefactor S.

GENERATE CHROMINANCE PIXELS IN SECOND COLOR SPACE process 1321 skips thereconstruction of the red and blue pixels that would normally be used inthe transformation to chrominance pixels. Instead, the transformationuses the filtered green-red color difference signals G.Rdiff and thefiltered green-blue color difference signals G.Bdiff for each locationin the captured frame 100. For example, direct computation from colordifference signals to Cb and Cr can be derived from the standardtransformation (1a)-(1c), with t11=65.481, t12=128.553, t13=24.966;t21=−37.797, t22=−74.203, t23=112; and t31=112, t32=−93.786,t33=−18.214:

$\begin{matrix}{{Cb}{= {{128} + {( {{112/2}55} )*{G.{Bdiff}}} + {( {37.79{7/2}55} )*{G.{Rdiff}}}}}} & ( {3a} ) \\{{Cb} = {128 + {( {{112/2}55} )*{G.{Bdiff}}} + {( {18.21{4/2}55} )*{G.{Bdiff}}}}} & ( {3b} )\end{matrix}$

Process 1321 applies this transformation to each location in thecaptured frame to generate the chrominance pixels at that location.However, the generation of the filtered green-red color differencesignals G.Rdiff and the filtered green-blue color difference signalsG.Bdiff is dependent upon the color of the captured pixel at thelocation.

In one aspect, the appropriate filtered green-red color differencesignals G.Rdiff and the filtered green-blue color difference signalsG.Bdiff are generated for a location and the above definition is used togenerate the chrominance pixels. GENERATE CHROMINANCE PIXELS IN SECONDCOLOR SPACE process 1321 can be implemented in a variety of ways. Thegeneration of the missing green pixels is most critical and so theprocess has higher spatial resolution. The detailed spatial resolutionis typically not required for the missing chrominance pixels and so asimpler reconstruction process can be used that uses color differencesignals already available during processes 303 and 304.

FIG. 14 is a more detailed process flow diagram of one implementation ofGENERATE CHROMINANCE PIXELS IN SECOND COLOR SPACE process 1321. Thisdiagram is illustrative only and is not intended to be limiting. Theillustrated process flow and the order of the process flow is used forease of discussion and is not intended to limit process 1321 to eitherthe specific acts or the specific sequence of acts shown. In view ofthis disclosure, one knowledgeable in the field can implement process1321 taking into consideration processor capability, available memory,timing requirements, etc.

Recall that FIG. 9 presented the data available at each location uponcompletion of process 305. FIGS. 15A to 15D are blocks taken from FIG. 9that represent the information used in this example of GENERATECHROMINANCE PIXELS IN SECOND COLOR SPACE process 1321.

For the example of GENERATE CHROMINANCE PIXELS IN SECOND COLOR SPACEprocess 1321 in FIG. 14, processing transfers from ENHANCE BRIGHTNESSPIXEL IN SECOND COLOR SPACE 1320 to RED PIXEL check process 1410. REDPIXEL check process 1410 determines whether the current pixel atlocation (x, y) is a red pixel R(x, y). If the current pixel at location(x, y) is a red pixel R(x, y), RED PIXEL check process 1410 transfers toGENERATE FILTERED GREEN-RED DIFFERENCE AND FILTERED GREEN-BLUEDIFFERENCE process 1411, sometimes referred to as process 1411, andotherwise transfers to BLUE PIXEL check process 1412.

GENERATE FILTERED GREEN-RED DIFFERENCE AND FILTERED GREEN-BLUEDIFFERENCE process 1411 generates filtered green-red color differencesignal G.Rdiff(x, y) and filtered green-blue color difference signalG.Bdiff(x, y) at location (x, y) of original red pixel R(x, y).Specifically, for a red pixel R(x, y), see FIG. 15A, filtered green-redcolor difference signal G.Rdiff(x, y) is generated as the average of thehorizontal filtered green-red color difference signal G.Rdiff_h(x, y)and vertical filtered green-red color difference signal G.Rdiff_v(x, y),i.e.,

G.Rdiff(x, y) = (G.Rdiff_h(x, y) + G.Rdiff_v(x, y))/2

In process 1411, the filtered green-blue color difference signalG.Bdiff(x, y) is generated as the average of the color green-blue colordifference signals at the four neighboring pixels, i.e.,

G.Bdiff(x, y) = (G_Bdiff_v(x − 1, y) + G_Bdiff_v(x + 1, y) + G_Bdiff_h(x, y − 1) + G_Bdiff_h(x, y + 1))/4

Upon completion, GENERATE FILTERED GREEN-RED DIFFERENCE AND FILTEREDGREEN-BLUE DIFFERENCE process 1411 transfers to GENERATE CHROMINANCEPIXELS process 1420A.

GENERATE CHROMINANCE PIXELS process 1420A uses filtered green-red colordifference signal G.Rdiff(x, y) and filtered green-blue color differencesignal G.Bdiff(x, y) at location (x, y) of original red pixel R(x, y) indefinitions (3a) and (3b) to generate chrominance pixels Cb and Cr atlocation (x, y). Upon completion GENERATE CHROMINANCE PIXELS process1420A transfers to LAST PIXEL check process 1414.

When RED PIXEL check process 1410 transfers to BLUE PIXEL check process1412, check process 1412 determines whether the current pixel atlocation (x, y) is a blue pixel B(x, y). If the current pixel atlocation (x, y) is a blue pixel B(x, y), BLUE PIXEL check process 1412transfers to GENERATE FILTERED GREEN-BLUE DIFFERENCE AND FILTEREDGREEN-BLUE DIFFERENCE process 1413, sometimes referred to as process1413, and otherwise transfers to LAST PIXEL check process 1414.

GENERATE FILTERED GREEN-BLUE DIFFERENCE AND FILTERED GREEN-REDDIFFERENCE process 1413 generates filtered green-blue color differencesignal G.Bdiff(x, y) and filtered green-red color difference signalG.Rdiff(x, y) at location (x, y) of original blue pixel B(x, y).Specifically, for a blue pixel B(x, y), see FIG. 15B, filteredgreen-blue color difference signal G.Bdiff(x, y) is generated as theaverage of the horizontal filtered green-blue color difference signalG.Bdiff_h(x, y) and vertical filtered green-blue color difference signalG.Bdiff_v(x, y), i.e.,

G.Bdiff(x, y) = (G.Bdiff_h(x, y) + G.Bdiff_v(x, y))/2

In process 1413, the filtered green-red color difference signalG.Rdiff(x, y) is generated as the average of the filtered green-redcolor difference signals at the four neighboring pixels, i.e.,

G.Rdiff(x, y) = (G_Rdiff_v(x − 1, y) + G_Rdiff_v(x + 1, y) + G_Rdiff_h(x, y + 1) + G_Rdiff_h(x, y + 1))/4

Upon completion, GENERATE FILTERED GREEN-BLUE DIFFERENCE AND FILTEREDGREEN-RED DIFFERENCE process 1413 transfers to GENERATE CHROMINANCEPIXELS process 1420A.

GENERATE CHROMINANCE PIXELS process 1420A uses filtered green-red colordifference signal G.Rdiff(x, y) and filtered green-blue color differencesignal G.Bdiff(x, y) at location (x, y) of original blue pixel B(x, y)in definitions (3a) and (3b) to generate chrominance pixels Cb and Cr atlocation (x, y). Upon completion GENERATE CHROMINANCE PIXELS process1420A transfers to LAST PIXEL check process 1414.

LAST PIXEL check process 1414 determines whether all the pixels in theframe have been processed. If all the pixels have been processed, LASTPIXEL check process 1414 resets the current pixel pointer to the startof the frame and then transfers to GREEN PIXEL check process 1415.Conversely, if all the pixels have not been processed, LAST PIXEL checkprocess 1414 adjusts a current pixel pointer to the next pixel and thentransfers to RED PIXEL check process 1410.

When LAST PIXEL check process 1414 transfers to GREEN PIXEL checkprocess 1415, GREEN PIXEL check process 1415 determines whether thecurrent pixel at location (x, y) is a green pixel G(x, y). If thecurrent pixel at location (x, y) is a green pixel G(x, y), GREEN PIXELcheck process 1415 transfers to BLUE-GREEN ROW check process 1416, andotherwise transfers to LAST PIXEL check process 1419.

The processing performed to generate filtered green-blue colordifference signal G.Bdiff(x, y) and filtered green-red color differencesignal G.Rdiff(x, y) at location (x, y) of original green pixel G(x, y)depends on the color of the pixels surrounding original green pixel G(x,y). Thus, BLUE-GREEN ROW check process 1416 determines whether originalgreen pixel G(x, y) is in a row of green and blue pixels. If originalgreen pixel G(x, y) is in a row of green and blue pixels, check process1416 transfers to a second GENERATE FILTERED GREEN-BLUE DIFFERENCE ANDFILTERED GREEN-RED DIFFERENCE process 1417, sometimes referred to asprocess 1417, and otherwise transfers to a third GENERATE FILTEREDGREEN-BLUE DIFFERENCE AND FILTERED GREEN-RED DIFFERENCE process 1418,sometimes referred to as process 1418.

When BLUE-GREEN ROW check process 1416 transfers to second GENERATEFILTERED GREEN-BLUE DIFFERENCE AND FILTERED GREEN-RED DIFFERENCE process1417, process 1417 generates filtered green-blue color difference signalG.Bdiff(x, y) and filtered green-red color difference signal G.Rdiff(x,y) at location (x, y) of original green pixel G(x, y). Filteredgreen-blue color difference signal G.Bdiff(x, y) is the average of thehorizontal and vertical filtered green-blue color difference signals atlocation (x, y), where the vertical filtered green-blue color differencesignal is the average of the filtered vertical green-blue colordifference signals at the four diagonal neighboring pixels. Filteredgreen-red color difference signal G.Rdiff(x, y) is the average of thehorizontal and vertical filtered green-red color difference signals atlocation (x, y), where the horizontal filtered green-red colordifference signals is the average of the filtered horizontal green-redcolor difference signals at the four diagonal neighboring pixels.

Specifically, for original green pixel G(x, y), see FIG. 15C, process1417 generates,

G.Bdiff(x, y) = G.Bdiff_h(x, y)/2 + (G.Bdiff_v(x − 1, y + 1) + G_Bdiff_v(x − 1, y − 1) + G_Bdiff_v(x + 1, y − 1) + G_Bdiff_v(x + 1, y + 1))/8andG.Rdiff(x, y) = G.Rdiff_v(x, y)/2 + G.Rdiff_h(x − 1, y + 1) + G_Rdiff_h(x − 1, y − 1) + G_Rdiff_h(x + 1, y − 1) + G_Rdiff_h(x + 1, y + 1))/8

Upon completion, second GENERATE FILTERED GREEN-BLUE DIFFERENCE ANDFILTERED GREEN-RED DIFFERENCE process 1417 transfers to GENERATECHROMINANCE PIXELS process 1420B.

GENERATE CHROMINANCE PIXELS process 1420B uses filtered green-red colordifference signal G.Rdiff(x, y) and filtered green-blue color differencesignal G.Bdiff(x, y) at location (x, y) of original green pixel G(x, y)in definitions (3a) and (3b) to generate chrominance pixels Cb and Cr atlocation (x, y). Upon completion GENERATE CHROMINANCE PIXELS process1420B transfers to LAST PIXEL check process 1419.

When BLUE-GREEN ROW check process 1416 transfers to third GENERATEFILTERED GREEN-BLUE DIFFERENCE AND FILTERED GREEN-RED DIFFERENCE process1418, process 1418 generates filtered green-blue color difference signalG.Bdiff(x, y) and filtered green-red color difference signal G.Rdiff(x,y) at location (x, y) of original green pixel G(x, y) (FIG. 15D).Filtered green-blue color difference signal G.Bdiff(x, y) is the averageof the horizontal and vertical filtered green-blue color differencesignals at location (x, y), where the horizontal filtered green-bluecolor difference signal is the average of the filtered verticalgreen-blue color difference signals at the four diagonal neighboringpixels. Filtered green-red color difference signal G.Rdiff(x, y) is theaverage of the horizontal and vertical filtered green-red colordifference signals at location (x, y), where the vertical filteredgreen-red color difference signals is the average of the filteredhorizontal green-red color difference signals at the four diagonalneighboring pixels.

Specifically, for original green pixel G(x, y), see FIG. 15D, process1418 generates,

G.Bdiff(x, y) = G.Bdiff_v(x, y)/2 + (G.Bdiff_h(x − 1, y + 1) + G_Bdiff_h(x − 1, y − 1) + G_Bdiff_h(x + 1, y − 1) + G_Bdiff_h(x + 1, y + 1))/8, andG.Rdiff(x, y) = G.Rdiff_h(x, y)/2 + G.Rdiff_v(x − 1, y + 1) + G_Rdiff_v(x − 1, y − 1) + G_Rdiff_v(x + 1, y − 1) + G_Rdiff_v(x + 1, y + 1))/8

Upon completion, second GENERATE FILTERED GREEN-BLUE DIFFERENCE ANDFILTERED GREEN-RED DIFFERENCE process 1418 transfers to GENERATECHROMINANCE PIXELS process 1420B.

GENERATE CHROMINANCE PIXELS process 1420B uses filtered green-red colordifference signal G.Rdiff(x, y) and filtered green-blue color differencesignal G.Bdiff(x, y) at location (x, y) of original green pixel G(x, y)in definitions (3a) and (3b) to generate chrominance pixels Cb and Cr atlocation (x, y). Upon completion GENERATE CHROMINANCE PIXELS process1420B transfers to LAST PIXEL check process 1419.

LAST PIXEL check process 1419 determines whether all the pixels in theframe have been processed. If all the pixels have been processed, LASTPIXEL check process 1419 transfers to SEND FULL RESOLUTION FRAME process312. Conversely, if all the pixels have not been processed, LAST PIXELcheck process 1419 adjusts a current pixel pointer to the next pixel andthen transfers to GREEN PIXEL check process 1415. When check process1419 transfers to SEND FULL RESOLUTION FRAME process 312, each locationin the frame has a complete set of Y, Cb, and Cr pixels.

As used herein, “first,” “second,” and “third” are adjectives used todistinguish between different color components and different elements.Thus, “first,” “second,” and “third” are not intended to imply anyordering of the components or the elements.

The above description and the accompanying drawings that illustrateaspects and embodiments of the present inventions should not be taken aslimiting—the claims define the protected inventions. Various mechanical,compositional, structural, electrical, and operational changes may bemade without departing from the spirit and scope of this description andthe claims. In some instances, well-known circuits, structures, andtechniques have not been shown or described in detail to avoid obscuringthe invention.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms—such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike—may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positions(i.e., locations) and orientations (i.e., rotational placements) of thedevice in use or operation in addition to the position and orientationshown in the figures. For example, if the device in the figures wereturned over, elements described as “below” or “beneath” other elementsor features would then be “above” or “over” the other elements orfeatures. Thus, the exemplary term “below” can encompass both positionsand orientations of above and below. The device may be otherwiseoriented (rotated 90 degrees or at other orientations) and the spatiallyrelative descriptors used herein interpreted accordingly. Likewise,descriptions of movement along and around various axes include variousspecial device positions and orientations.

The singular forms “a”, “an”, and “the” are intended to include theplural forms as well, unless the context indicates otherwise. The terms“comprises”, “comprising”, “includes”, and the like specify the presenceof stated features, steps, operations, elements, and/or components butdo not preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups. Componentsdescribed as coupled may be electrically or mechanically directlycoupled, or they may be indirectly coupled via one or more intermediatecomponents.

Memory refers to a volatile memory, a non-volatile memory, or anycombination of the two. A processor is coupled to a memory containinginstructions executed by the processor. This could be accomplishedwithin a computer system, or alternatively via a connection to anothercomputer via modems and analog lines, or digital interfaces and adigital carrier line.

Herein, a computer program product comprises a non-transitory mediumconfigured to store computer readable code needed for any one or anycombination of the operations described with respect to the demosaicingprocess or in which computer readable code for any one or anycombination of operations described with respect to the demosaicingprocess is stored. Some examples of computer program products are CD-ROMdiscs, DVD discs, flash memory, ROM cards, floppy discs, magnetic tapes,computer hard drives, servers on a network and signals transmitted overa network representing computer readable program code. A non-transitorytangible computer program product comprises a non-transitory tangiblemedium configured to store computer readable instructions for any oneof, or any combination of operations described with respect to thedemosaicing process or in which computer readable instructions for anyone of, or any combination of operations described with respect to thedemosaicing process are stored. Non-transitory tangible computer programproducts are CD-ROM discs, DVD discs, flash memory, ROM cards, floppydiscs, magnetic tapes, computer hard drives and other non-transitoryphysical storage mediums.

In view of this disclosure, instructions used in any one of, or anycombination of operations described with respect to the enhancedemosaicing method can be implemented in a wide variety of computersystem configurations using an operating system and computer programminglanguage of interest to the user.

All examples and illustrative references are non-limiting and should notbe used to limit the claims to specific implementations and embodimentsdescribed herein and their equivalents. The headings are solely forformatting and should not be used to limit the subject matter in anyway, because text under one heading may cross reference or apply to textunder one or more headings. Finally, in view of this disclosure,particular features described in relation to one aspect or embodimentmay be applied to other disclosed aspects or embodiments of theinvention, even though not specifically shown in the drawings ordescribed in the text.

What is claimed is:
 1. A method comprising: receiving, by a demosaicmodule, a frame of raw pixels captured by an image capture sensor in animage capture unit having a color filter array, each location in theframe of raw pixels including one captured color pixel value and missingtwo color pixel values of a first color space; generating, by thedemosaic module for each location in the frame of raw pixels, a verticalcolor difference signal and a horizontal color difference signal usinginformation from the frame of raw pixels; generating, by the demosaicmodule for each location in the frame of raw pixels that is missing acaptured color pixel value of a first color component of the first colorspace, a reconstructed color pixel value of the first color componentsuch that a captured color pixel value or a reconstructed color pixelvalue of the first color component is known for each location in theframe of raw pixels; transforming, by the demosaic module for eachlocation in the frame of raw pixels, the captured color pixel value orthe reconstructed color pixel value of the first color component of therespective location to a brightness pixel value in a second color spacethat is different from the first color space; and transforming, by thedemosaic module for each location in the frame of raw pixels, thevertical color difference signal and the horizontal color differencesignal of the respective location into chrominance pixel values in thesecond color space.
 2. The method of claim 1, wherein transforming thevertical color difference signal and the horizontal colors differencesignal of the respective location into chrominance pixel values in thesecond color space is performed without using the brightness pixel valuein the second color space.
 3. The method of claim 1, whereintransforming the vertical color difference signal and the horizontalcolors difference signal of the respective location into chrominancepixel values in the second color space comprises: determining an averageof the vertical color difference signal and the horizontal colorsdifference signal; and using the average of the vertical colordifference signal and the horizontal colors difference signal todetermine the chrominance pixel values in the second color space.
 4. Themethod of claim 1, wherein for a particular location: the captured colorpixel value is a captured green pixel value; the horizontal colordifference signal is the captured green pixel value minus a horizontalestimated blue pixel value; and the vertical color difference signal isthe captured green pixel value minus a vertical estimated red pixelvalue.
 5. The method of claim 1, wherein transforming the captured colorpixel value or the reconstructed color pixel value of the first colorcomponent of the respective location to the brightness pixel in thesecond color space for each location in the frame of raw pixels isperformed without using a color pixel value of a second color componentof the first color space.
 6. The method of claim 5, wherein transformingthe captured color pixel value or the reconstructed color pixel value ofthe first color component of the respective location to the brightnesspixel in the second color space for each location in the frame of rawpixels is performed without using a color pixel value of a third colorcomponent of the first color space.
 7. The method of claim 1, furthercomprising: generating, by the demosaic module for each location in theframe of raw pixels, a vertical estimated color pixel value and ahorizontal estimated color pixel value using information from the frameof raw pixels, wherein for a location in the frame of raw pixels havinga captured color pixel value of the second color component of the firstcolor space, the vertical estimated color pixel value is of the firstcolor component of the first color space, and the horizontal estimatedcolor pixel value is of the first color component of the first colorspace; wherein generating the vertical color difference signal and thehorizontal color difference signal for each location in the frame ofpixels uses the vertical estimated color pixel value and the horizontalestimated color pixel value, wherein for the location in the frame ofraw pixels having the captured color pixel value of the second colorcomponent of the first color space, the vertical color difference signalis generated based on the vertical estimated color pixel value of thefirst color component of the first color space, and the horizontal colordifference signal is generated based on the horizontal estimated colorpixel value of the first color component of the first color space.
 8. Asystem comprising: a processor; and a memory communicatively coupled tothe processor and storing instructions executable by the processor to:receive a frame of raw pixels captured by an image capture sensor in animage capture unit having a color filter array, each location in theframe of raw pixels including one captured color pixel value and missingtwo color pixel values of a first color space; generate, for eachlocation in the frame of raw pixels, a vertical color difference signaland a horizontal color difference signal using information from theframe of raw pixels; generate, for each location in the frame of rawpixels that is missing a captured color pixel value of a first colorcomponent of the first color space, a reconstructed color pixel value ofthe first color component such that a captured color pixel value or areconstructed color pixel value of the first color component is knownfor each location in the frame of raw pixels; transform, for eachlocation in the frame of raw pixels, the captured color pixel value orthe reconstructed color pixel value of the first color component of therespective location to a brightness pixel value in a second color spacethat is different from the first color space; and transform, for eachlocation in the frame of raw pixels, the vertical color differencesignal and the horizontal color difference signal of the respectivelocation into chrominance pixel values in the second color space.
 9. Thesystem of claim 8, wherein transforming the vertical color differencesignal and the horizontal colors difference signal of the respectivelocation into chrominance pixel values in the second color space isperformed without using the brightness pixel value in the second colorspace.
 10. The system of claim 8, wherein transforming the verticalcolor difference signal and the horizontal colors difference signal ofthe respective location into chrominance pixel values in the secondcolor space comprises: determining an average of the vertical colordifference signal and the horizontal colors difference signal; and usingthe average of the vertical color difference signal and the horizontalcolors difference signal to determine the chrominance pixel values inthe second color space.
 11. The system of claim 8, wherein for aparticular location: the captured color pixel value is a captured greenpixel value; the horizontal color difference signal is the capturedgreen pixel value minus a horizontal estimated blue pixel value; and thevertical color difference signal is the captured green pixel value minusa vertical estimated red pixel value.
 12. The system of claim 8, whereintransforming the captured color pixel value or the reconstructed colorpixel value of the first color component of the respective location tothe brightness pixel in the second color space for each location in theframe of raw pixels is performed without using a color pixel value of asecond color component of the first color space.
 13. The system of claim12, wherein transforming the captured color pixel value or thereconstructed color pixel value of the first color component of therespective location to the brightness pixel in the second color spacefor each location in the frame of raw pixels is performed without usinga color pixel value of a third color component of the first color space.14. The system of claim 8, the instructions further executable by theprocessor to: generate, for each location in the frame of raw pixels, avertical estimated color pixel value and a horizontal estimated colorpixel value using information from the frame of raw pixels, wherein fora location in the frame of raw pixels having a captured color pixelvalue of the second color component of the first color space, thevertical estimated color pixel value is of the first color component ofthe first color space, and the horizontal estimated color pixel value isof the first color component of the first color space; whereingenerating the vertical color difference signal and the horizontal colordifference signal for each location in the frame of pixels uses thevertical estimated color pixel value and the horizontal estimated colorpixel value, wherein for the location in the frame of raw pixels havingthe captured color pixel value of the second color component of thefirst color space, the vertical color difference signal is generatedbased on the vertical estimated color pixel value of the first colorcomponent of the first color space, and the horizontal color differencesignal is generated based on the horizontal estimated color pixel valueof the first color component of the first color space.
 15. Anon-transitory computer-readable medium storing instructions executableby a processor to: receive a frame of raw pixels captured by an imagecapture sensor in an image capture unit having a color filter array,each location in the frame of raw pixels including one captured colorpixel value and missing two color pixel values of a first color space;generate, for each location in the frame of raw pixels, a vertical colordifference signal and a horizontal color difference signal usinginformation from the frame of raw pixels; generate, for each location inthe frame of raw pixels that is missing a captured color pixel value ofa first color component of the first color space, a reconstructed colorpixel value of the first color component such that a captured colorpixel value or a reconstructed color pixel value of the first colorcomponent is known for each location in the frame of raw pixels;transform, for each location in the frame of raw pixels, the capturedcolor pixel value or the reconstructed color pixel value of the firstcolor component of the respective location to a brightness pixel valuein a second color space that is different from the first color space;and transform, for each location in the frame of raw pixels, thevertical color difference signal and the horizontal color differencesignal of the respective location into chrominance pixel values in thesecond color space.
 16. The non-transitory computer-readable medium ofclaim 15, wherein transforming the vertical color difference signal andthe horizontal colors difference signal of the respective location intochrominance pixel values in the second color space is performed withoutusing the brightness pixel value in the second color space.
 17. Thenon-transitory computer-readable medium of claim 15, whereintransforming the vertical color difference signal and the horizontalcolors difference signal of the respective location into chrominancepixel values in the second color space comprises: determining an averageof the vertical color difference signal and the horizontal colorsdifference signal; and using the average of the vertical colordifference signal and the horizontal colors difference signal todetermine the chrominance pixel values in the second color space. 18.The non-transitory computer-readable medium of claim 15, wherein for aparticular location: the captured color pixel value is a captured greenpixel value; the horizontal color difference signal is the capturedgreen pixel value minus a horizontal estimated blue pixel value; and thevertical color difference signal is the captured green pixel value minusa vertical estimated red pixel value.
 19. The non-transitorycomputer-readable medium of claim 15, wherein transforming the capturedcolor pixel value or the reconstructed color pixel value of the firstcolor component of the respective location to the brightness pixel inthe second color space for each location in the frame of raw pixels isperformed without using a color pixel value of a second color componentof the first color space.
 20. The non-transitory computer-readablemedium of claim 19, wherein transforming the captured color pixel valueor the reconstructed color pixel value of the first color component ofthe respective location to the brightness pixel in the second colorspace for each location in the frame of raw pixels is performed withoutusing a color pixel value of a third color component of the first colorspace.